Improving Search Effectiveness in the Legal E-Discovery Process Using Relevance Feedback

Finding information and preserving confidentiality are in some sense opposite goals, but there are a number of practical situations in which a balance must be struck between the two. Identifying relevant evidence in large collections of digital business records, the so-called “e-discovery” problem, is one such situation. Present practice involves consultation between attorneys representing plaintiffs and defendants, a search conducted by defendants (or their agents) in response to plaintiffs’ requests, manual review of every retrieved document, and release to the plaintiffs of all documents that are judged relevant and not subject to a claim of privilege. Although advanced search strategies and tools are available, the keyword based search dominates current legal practice in e-discovery as it is well understood and has been commonly used by the legal community for a long time. However, it is difficult for a party to select the right keywords to achieve a satisfying recall level without knowledge of the other party’s data. This paper applies relevance feedback and result fusion techniques as a simple model for a multi-stage consultation process. Experiments using the TREC Robust Track test collection and relevance feedback using offer weights show that a partial release of relevant documents, followed by a second consultation, has the potential to substantially improve overall retrieval effectiveness, and that additional partial releases and subsequent consultations seem to offer diminishing potential for additional benefit.